Deep Perceptual Mapping for Thermal to Visible Face Recogntion

Cross modal face matching between the thermal and visible spectrum is a much desired capability for night-time surveillance and security applications. Due to a very large modality gap, thermal-to-visible face recognition is one of the most challenging face matching problem. In this paper, we present an approach to bridge this modality gap by a significant margin. Our approach captures the highly non-linear relationship between the two modalities by using a deep neural network. Our model attempts to learn a non-linear mapping from visible to thermal spectrum while preserving the identity information. We show substantive performance improvement on a difficult thermal-visible face dataset (UND-X1). The presented approach improves the state-of-the-art by more than 10% in terms of Rank-1 identification and bridge the drop in performance due to the modality gap by more than 40%. The goal of training the deep network is to learn the projections that can be used to bring the two modalities together. Typically, this would mean regressing the representation from one modality towards the other. We construct a deep network comprising N +1 layers with m(k) units in the k-th layer, where k = 1,2, · · · ,N. For an input of x ∈Rd , each layer will output a non-linear projection by using the learned projection matrix W and the non-linear activation function g(·). The output of the k-th hidden layer is h(k) = g(W(k)h(k−1) + b(k)), where W(k) ∈ Rm×m(k−1) is the projection matrix to be learned in that layer, b(k) ∈Rm is a bias vector and g : Rm 7→ Rm is the non-linear activation function. Similarly, the output of the most top level hidden layer can be computed as: